Skip to content

Avoid redundant scans when extracting stage0 archives - #165819

Open
TaKO8Ki wants to merge 2 commits into
rust-lang:mainfrom
TaKO8Ki:avoid-stage0-archive-rescans
Open

TaKO8Ki wants to merge 2 commits into
rust-lang:mainfrom
TaKO8Ki:avoid-stage0-archive-rescans

Conversation

@TaKO8Ki

@TaKO8Ki TaKO8Ki commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

getnames() calls getmembers(), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each TarInfo directly to extract() also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, RustBuild.download_toolchain() improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 11, 2026
@rustbot

rustbot commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: bootstrap
  • bootstrap expanded to 6 candidates
  • Random selection from Mark-Simulacrum, clubby789, jieyouxu

@Mark-Simulacrum

Copy link
Copy Markdown
Member

@bors r+ rollup

@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

📌 Commit ce09ca6 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 12, 2026
Zalathar added a commit to Zalathar/rust that referenced this pull request Nov 12, 2026
…ns, r=Mark-Simulacrum

Avoid redundant scans when extracting stage0 archives

[`getnames()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1991-L1995) calls [`getmembers()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1981-L1989), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each `TarInfo` directly to [`extract()`](https://docs.python.org/3.10/library/tarfile.html#tarfile.TarFile.extract) also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, `RustBuild.download_toolchain()` improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.
rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Rollup of 6 pull requests

Successful merges:

 - #165778 (Bump stage0 to 1.101.0-beta.1)
 - #163077 (feat: support nested functions with `#[offload_kernel]`)
 - #165578 (Avoid `Hash::hash` when stable-hashing various types)
 - #165729 (tests: Add regression test for infinite recursive dereference)
 - #165819 (Avoid redundant scans when extracting stage0 archives)
 - #165822 (Remove unused compiler modules and types)
Zalathar added a commit to Zalathar/rust that referenced this pull request Nov 12, 2026
…ns, r=Mark-Simulacrum

Avoid redundant scans when extracting stage0 archives

[`getnames()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1991-L1995) calls [`getmembers()`](https://github.com/python/cpython/blob/b4e48a444ea02921ce4b701fe165e6cfd4cf5845/Lib/tarfile.py#L1981-L1989), which can scan the entire archive if its members have not already been loaded. Iterating over tar directly (https://github.com/python/cpython/blob/v3.10.12/Lib/tarfile.py#L2715-L2747) allows archive entries to be handled one at a time. Passing each `TarInfo` directly to [`extract()`](https://docs.python.org/3.10/library/tarfile.html#tarfile.TarFile.extract) also avoids name-based lookup, removing the preliminary scan and its associated repeated decompression.

With cached stage0 archives, `RustBuild.download_toolchain()` improved from 6.44 s to 3.50 s (medians of two runs per version). The rustc, rust-std, and cargo archives were extracted in parallel into fresh directories. Output hashes were verified separately.
rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Rollup of 12 pull requests

Successful merges:

 - #165804 (cg_llvm: Use an invariant lifetime brand for LLVM values)
 - #165778 (Bump stage0 to 1.101.0-beta.1)
 - #165412 (Implement `core::ops::FnAddr`)
 - #165487 (Migrate built-in macros `pattern_type` & `view_type` to macros 2.0 that expand to internal syntax)
 - #165818 (do not lint `redundant_lifetimes` on items from external macros)
 - #163077 (feat: support nested functions with `#[offload_kernel]`)
 - #164795 (Extend support of lint `explicit_outlives_requirements` to checked type aliases)
 - #165578 (Avoid `Hash::hash` when stable-hashing various types)
 - #165729 (tests: Add regression test for infinite recursive dereference)
 - #165819 (Avoid redundant scans when extracting stage0 archives)
 - #165822 (Remove unused compiler modules and types)
 - #165824 (Add regression test for HRTB projection bound with TryFrom error)
@Zalathar

Copy link
Copy Markdown
Member

@bors r-
@bors try jobs=dist-x86_64-msvc

@rust-bors rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 12, 2026
@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request was unapproved.

This PR was contained in a rollup (#165843), which was unapproved.

View changes since this unapproval

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Avoid redundant scans when extracting stage0 archives


try-job: dist-x86_64-msvc
@Zalathar

Copy link
Copy Markdown
Member

Looks like the CI failure was probably unrelated flakiness, though I'm marking this as iffy in case this PR is the cause of the flakiness.

@bors r=Mark-Simulacrum rollup=iffy

@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

📌 Commit ce09ca6 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 12, 2026
@TaKO8Ki

TaKO8Ki commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@bors r-

@rust-bors rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 12, 2026
@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request was unapproved.

View changes since this unapproval

@TaKO8Ki

TaKO8Ki commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

This PR does not appear to have introduced the underlying race condition. However, the change in extraction timing may have exposed an existing intermittent failure. I will check it.

@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 61389fc (61389fc7cda28f5f00a85f268c7165011bf8d4ab)
Base parent: ee8268e (ee8268e23d0e2e308b52e4f63df69afb69bc61a4)

@TaKO8Ki
TaKO8Ki force-pushed the avoid-stage0-archive-rescans branch from ce09ca6 to 832e0f3 Compare November 12, 2026 08:41
@rustbot

rustbot commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

@TaKO8Ki

TaKO8Ki commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@bors try jobs=dist-x86_64-msvc

@rust-bors

rust-bors Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

⌛ Trying commit 832e0f3 with merge b06d48c…

To cancel the try build, run the command @bors try cancel.

Workflow: https://github.com/rust-lang/rust/actions/runs/43185061034

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
Avoid redundant scans when extracting stage0 archives


try-job: dist-x86_64-msvc

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants